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To  provide  timely,  accurate  translation  services 
for  the  security  of  the  nation 
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•  Translation 

-  English  to  target  language(s) 

-  Foreign  language(s)  to  target 
language(s)  -  usually  English 

•  Transcription 

-  Audio 

-  Video 

•  Summarization 

•  Gisting 
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Translated  for  approx.  30 
U.S.  Government  entities 


Translated  from  more  than  50 
Languages 

Translated  into  more  than  70 
languages 
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TRANSLATION 


Examples: 

•  Book  chapters 

•  Journal  articles 

•  Web  news  articles 

•  Email  messages 

•  PowerPoint  slides 

•  Brochures 

•  TV  broadcasts 

•  Radio  broadcasts 

•  Maps 


Government  User  Session 


Motivations  for  TM 
Technology  Assessment 


•  Application  of  TM  technology 
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-  Industry 

•  Automobile  industry  has  documented  return  on  investment  for 
updating  the  translation  of  auto  manufacturing  manuals  for  off¬ 
shore  assembly  plants 

•  Standardized  terminology  for  Parts  and  Components  Suppliers 


-  Example 

•  General  Motors 


Languages:  English  to  German,  European  Spanish,  Canadian 
French,  Italian,  Mexican  Spanish  and  Dutch 
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Motivations  for  TM 
Technology  Assessment 
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*  NVTC  also  translates  manuals: 

-  U.S.  Army  field  manuals  from 
English  to  Arabic  and  other 
languages  to  train  Iraqi  Security 
Forces  and  other  coalition  partners. 


•  NVTC  translates  many  other 
genres  as  well: 


-  Journal  articles,  news  stories, 
broadcast  interviews,  memoranda, 
emails,  web  sites,  PowerPoint 
slides,  etc. 
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Goals: 


Translation  Memory 
Technology  Assessment:  Pilot 
Study 
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•  Develop  TM  assessment  methodology  using  3  common  languages 

•  Develop  candidate  performance  metrics  for  assessing  TM  usage 
across  genres  (e.g.,  translation  speed,  number  of  QC  edits) 

•  Develop  candidate  linguistic  parameters  to  characterize  genres 
(e.g.,  sentence  structure,  morphology) 

•  Initiate  development  of  TM  memory  banks  for  3  languages 


End  goal: 

•  To  determine  if  and  under  what  circumstances  using  TM 

technology  will  result  in  improvements  in  translation  speed  and 
accuracy  when  translating  1C  and  DoD  documents  in  the  NVTC 
workflow 
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•  NVTC 


NVTC  -  Naval  Research  Laboratory 
Joint  Research  Partnership 
to  Assess  TM  Technology 


-  Data  for  translation 

-  Translators,  quality  control  professionals 

-  Researchers 
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NRL 

-  Data  for  translation 

-  Researchers 

-  Joint  Research  Agreement  (in  progress) 

•  First  NRL  joint  agreement  with  the  1C 

•  First  NVTC  joint  agreement  with  US  Government  research  entity 

NIST 


-  Assessment  of  experimental  methods 

-  IC/DoD  Workshops 

-  Metrics  development 


NIST 
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Technology  Assessment:  Pilot 
Study 


NATIONAL 

VIRTUAL 


TRANSLATION 

CENTER 


Languages 

Documents 

Memory  Bank  Source  Data 

Arabic  and  Chinese 

Transcripts  of  broadcast  news 

Transcripts  of  broadcast  news 

Russian 

Military  journal  articles 

Unknown  source 

For  each  language: 

•  One  control  document  translated  by  hand 

•  Three  test  documents  translated  using  TM  tool 

•  Two  translators  each  translate  four  documents  (control  and  test) 


•  Two  QC  professionals  check  the  four  translations,  one  QC  professional 
per  translator 
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Analysis 

Compare 


Translation  Memory 
Technology  Assessment:  Pilot 
Study 
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•  The  quality  of  the  translation  for  Arabic  and  Chinese.  &  44  44  44 

-  Will  highlight  what  linguistic  criteria  (e.g.,  sentence  ||  5 

structure,  morphology)  might  be  relevant  in  characterizim  £*  I*-**  ** 

genres.  V-  4  4 

Compare 

•  The  quality  of  the  Russian  translations  to  the  Arabic 

and  Chinese  translations.  . .  M... 


-  Chinese  and  Arabic  will  show  the  effect  of  using  a  memory 
bank  built  from  documents  of  the  same  genre  as  the 
translation  source  documents 

-  Russian  will  show  the  effect  of  using  a  memory  bank  that 
is  populated  with  unrelated  genre  segments. 
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Compare 

•  The  control  document  translated  “by  hand”  to  the 
translation  of  the  matching  test  document  (comparable 
genre,  length,  and  topic)  using  the  TM  tool. 

Will  give  some  measure  of  how  much  TM  technology  helps 
improve  translation  speed  and  accuracy  for  each  language 
and  genre  combination. 

Government  User  Session 


IrAn  ci  nvf\f rr vtt 


- 


What  did  we  Measure? 
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•  Translator: 

-  Usability  (quantitative) 

•  Mouse  clicks 

•  Tool  use  log 

•  Memory  bank  updates 

-  User  assessment  (qualitative) 

•  Questionnaires 

-  After  manual  and  TM-assisted  tasks 

-  Controls: 

•  Used  single  (hardcopy)  dictionary  per  language 

•  Trained  on  TM  system 


•  Quality  Control  Professional: 


User  assessment  (qualitative) 

•  Questionnaires  on  each  translation  QC’ed 
Controls:  not  told  which  translations  were  purely  manual 
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Apart  from  free-response  prompts,  all  questionnaire  items  required  the  user  to 
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anslators  made  extremely  heavy  use  of  the  automatic  spell-checker. 

We  did  not  anticipate  this  behavior.  - iranslat^on 
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Russian-English  translation. 
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Chinese-English  translation. 
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Software  Evaluation 
Questionnaire:  Conclusions 
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•  Ignoring  differences  between  sessions,  users  seemed  to  have  a 
slightly  favorable  opinion  of  the  TM  system’s: 

-  Utility 

-  ease  of  use 

-  Information  retrieval  performance. 

•  This  data  is  anecdotal,  but  suggests  a  trend: 

-  Subjects’  opinion  of  the  system  generally  improved  in  later  sessions 

-  Further  study  with  a  larger  group  of  subjects  may  produce 
statistically  significant  results. 
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-  Translators  are  used  to  multiple  internet  sources,  colleagues, 
specialized  glossaries  for  terminology  support 

•  Limiting  terminology  source  intended  as  control,  but  creates 
potential  bias 

•  Lesson  learned:  allow  more  access  to  open  sources 

-  Translators  improved  use  of  TM  quickly  over  a  few  days 

•  Lesson  learned:  longer  ramp-up  time 

-  Native  memory  bank  (the  Russian  memory  bank  resident  in  the 
commercial  TM)  was  insufficient  for  Russian  military  journal 
articles 

•  Rarely  scored  matches  of  significant  length  *  4  c  M  h  t  b  * 

•  Lesson  learned:  native  memory  bank  probably  not  sufficient  for 
domain  and/or  genre  specific  text. 
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Collaboration  with  the  National 
Institute  of  Standards  and 
Technology  (NIST) 

•  NVTC-NRL  will  present  the  pilot  study  TM  assessment 
methodology  and  results  to  NIST. 

•  NIST  will  validate  and  refine  the  outcomes  from  the  pilot  study, 
resulting  in: 

-  A  set  of  candidate  performance  metrics  for  measuring  the  translation 
improvement  when  using  TM  technology; 

-  A  series  of  workshops  for  researchers,  developers,  users,  and  other 
TM  stakeholders; 

-  An  augmentation  of  the  MetricsMATR  series,  or  similar  forum  for  TM 
metrics 
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Meanwhile,  NVTC-NRL  will  begin  a  memory  bank  for  a  4th  language 
(possibly  from  Africa)  to  use  in  follow-up  assessment 
experiments. 
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Follow-on  Study 


Translation  Memory 
Technology  Assessment: 
Follow-On  Study 
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Continue  development  of  memory  banks  for  all 
languages 

-  Arabic,  Chinese,  Russian,  African  language 
Add  new  genres 
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Increase  independence  of  experimental  components 


-  Pilot  study  measures  effects  of  TM  technology  for 
assisting  translators  as  individuals. 


-  Follow-on  study  will  better  distinguish  between  the 
translator  and  the  TM  technology  as  the  direct  source  of 
improvement 


•  Study  the  structure  and  use  of  memory  banks 


Investigate  organizing  memory  banks  by  genre,  topic, 
linguistic  structure,  or  other  criteria 
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